Businesses like banks that provide service have to worry about the problem of 'Churn' i.e. customers leaving and joining another service provider. It is important to understand which aspects of the service influence a customer's decision in this regard. Management can concentrate efforts on the improvement of service, keeping in mind these priorities.
Given a Bank customer, build a neural network-based classifier that can determine whether they will leave or not in the next 6 months.
The case study is from an open-source dataset from Kaggle. The dataset contains 10,000 sample points with 14 distinct features such as CustomerId, CreditScore, Geography, Gender, Age, Tenure, Balance, etc
#importing tensorflow
import tensorflow as tf
print(tf.__version__)
2.5.0
import warnings
warnings.filterwarnings("ignore")
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
import tensorflow as tf
from sklearn import preprocessing
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout
from sklearn.metrics import accuracy_score, confusion_matrix, precision_score, recall_score, f1_score, precision_recall_curve, auc
import matplotlib.pyplot as plt
from tensorflow.keras import optimizers
from sklearn.decomposition import PCA
import seaborn as sns
import keras
import tensorflow as tf
from keras import backend as K
from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.optimizers import Adam
from sklearn.preprocessing import StandardScaler
from google.colab import drive
drive.mount('/content/drive')
Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
#Defining the path of the dataset
project_path = '/content/drive/My Drive/AIML/Project6/'
dataset_file = project_path + 'bank.csv'
churn_df=pd.read_csv (dataset_file) #read in data
churn_df.sample(5)
| RowNumber | CustomerId | Surname | CreditScore | Geography | Gender | Age | Tenure | Balance | NumOfProducts | HasCrCard | IsActiveMember | EstimatedSalary | Exited | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 5987 | 5988 | 15636634 | Lindon | 630 | Germany | Female | 25 | 7 | 79656.81 | 1 | 1 | 0 | 93524.22 | 0 |
| 3169 | 3170 | 15688172 | Tai | 677 | Spain | Male | 40 | 5 | 0.00 | 2 | 1 | 0 | 88947.56 | 0 |
| 5562 | 5563 | 15614361 | Liao | 620 | Spain | Male | 42 | 9 | 121490.05 | 1 | 1 | 1 | 29296.74 | 0 |
| 6554 | 6555 | 15632576 | Yashina | 520 | France | Male | 31 | 4 | 93249.40 | 1 | 1 | 0 | 77335.75 | 0 |
| 1859 | 1860 | 15681956 | Bailey | 684 | France | Male | 34 | 9 | 0.00 | 2 | 1 | 1 | 65257.57 | 0 |
Insights:
Check Shape of Data, Check data types and number of non-null values for each column.
churn_df.shape
(10000, 14)
Insights:
churn_df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 10000 entries, 0 to 9999 Data columns (total 14 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 RowNumber 10000 non-null int64 1 CustomerId 10000 non-null int64 2 Surname 10000 non-null object 3 CreditScore 10000 non-null int64 4 Geography 10000 non-null object 5 Gender 10000 non-null object 6 Age 10000 non-null int64 7 Tenure 10000 non-null int64 8 Balance 10000 non-null float64 9 NumOfProducts 10000 non-null int64 10 HasCrCard 10000 non-null int64 11 IsActiveMember 10000 non-null int64 12 EstimatedSalary 10000 non-null float64 13 Exited 10000 non-null int64 dtypes: float64(2), int64(9), object(3) memory usage: 1.1+ MB
Insights:
for feature in churn_df.columns: # Loop through all columns in the dataframe
if churn_df[feature].dtype == 'object': # Only apply for columns with categorical strings
churn_df[feature] = pd.Categorical(churn_df[feature])# Change to categorical varible
churn_df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 10000 entries, 0 to 9999 Data columns (total 14 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 RowNumber 10000 non-null int64 1 CustomerId 10000 non-null int64 2 Surname 10000 non-null category 3 CreditScore 10000 non-null int64 4 Geography 10000 non-null category 5 Gender 10000 non-null category 6 Age 10000 non-null int64 7 Tenure 10000 non-null int64 8 Balance 10000 non-null float64 9 NumOfProducts 10000 non-null int64 10 HasCrCard 10000 non-null int64 11 IsActiveMember 10000 non-null int64 12 EstimatedSalary 10000 non-null float64 13 Exited 10000 non-null int64 dtypes: category(3), float64(2), int64(9) memory usage: 1001.7 KB
for feature in churn_df.columns:
if churn_df[feature].name in churn_df.select_dtypes(include='category').columns:
print(churn_df[feature].value_counts())
print('-'*30)
Smith 32
Martin 29
Scott 29
Walker 28
Brown 26
..
Perreault 1
Perkin 1
Percy 1
Hardacre 1
Fancher 1
Name: Surname, Length: 2932, dtype: int64
------------------------------
France 5014
Germany 2509
Spain 2477
Name: Geography, dtype: int64
------------------------------
Male 5457
Female 4543
Name: Gender, dtype: int64
------------------------------
Insights:
churn_df.isna().sum()
RowNumber 0 CustomerId 0 Surname 0 CreditScore 0 Geography 0 Gender 0 Age 0 Tenure 0 Balance 0 NumOfProducts 0 HasCrCard 0 IsActiveMember 0 EstimatedSalary 0 Exited 0 dtype: int64
Insights:
churn_df.nunique()
RowNumber 10000 CustomerId 10000 Surname 2932 CreditScore 460 Geography 3 Gender 2 Age 70 Tenure 11 Balance 6382 NumOfProducts 4 HasCrCard 2 IsActiveMember 2 EstimatedSalary 9999 Exited 2 dtype: int64
Insights:
churn_df.describe()
| RowNumber | CustomerId | CreditScore | Age | Tenure | Balance | NumOfProducts | HasCrCard | IsActiveMember | EstimatedSalary | Exited | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| count | 10000.00000 | 1.000000e+04 | 10000.000000 | 10000.000000 | 10000.000000 | 10000.000000 | 10000.000000 | 10000.00000 | 10000.000000 | 10000.000000 | 10000.000000 |
| mean | 5000.50000 | 1.569094e+07 | 650.528800 | 38.921800 | 5.012800 | 76485.889288 | 1.530200 | 0.70550 | 0.515100 | 100090.239881 | 0.203700 |
| std | 2886.89568 | 7.193619e+04 | 96.653299 | 10.487806 | 2.892174 | 62397.405202 | 0.581654 | 0.45584 | 0.499797 | 57510.492818 | 0.402769 |
| min | 1.00000 | 1.556570e+07 | 350.000000 | 18.000000 | 0.000000 | 0.000000 | 1.000000 | 0.00000 | 0.000000 | 11.580000 | 0.000000 |
| 25% | 2500.75000 | 1.562853e+07 | 584.000000 | 32.000000 | 3.000000 | 0.000000 | 1.000000 | 0.00000 | 0.000000 | 51002.110000 | 0.000000 |
| 50% | 5000.50000 | 1.569074e+07 | 652.000000 | 37.000000 | 5.000000 | 97198.540000 | 1.000000 | 1.00000 | 1.000000 | 100193.915000 | 0.000000 |
| 75% | 7500.25000 | 1.575323e+07 | 718.000000 | 44.000000 | 7.000000 | 127644.240000 | 2.000000 | 1.00000 | 1.000000 | 149388.247500 | 0.000000 |
| max | 10000.00000 | 1.581569e+07 | 850.000000 | 92.000000 | 10.000000 | 250898.090000 | 4.000000 | 1.00000 | 1.000000 | 199992.480000 | 1.000000 |
Insights:
churn_df.describe(include=['category']).T
| count | unique | top | freq | |
|---|---|---|---|---|
| Surname | 10000 | 2932 | Smith | 32 |
| Geography | 10000 | 3 | France | 5014 |
| Gender | 10000 | 2 | Male | 5457 |
Insights:
#Function to create Histogram and Boxplot
def histogram_boxplot(feature, figsize=(10,5), bins = None):
""" Boxplot and histogram combined
feature: 1-d feature array
figsize: size of fig (default (9,8))
bins: number of bins (default None / auto)
"""
f2, (ax_box2, ax_hist2) = plt.subplots(nrows = 2,
sharex = True,
gridspec_kw = {"height_ratios": (.25, .75)},
figsize = figsize
)
sns.boxplot(feature, ax=ax_box2, showmeans=True, color='violet')
sns.distplot(feature, kde=F, ax=ax_hist2, bins=bins,palette="winter") if bins else sns.distplot(feature, kde=False, ax=ax_hist2)
ax_hist2.axvline(np.mean(feature), color='green', linestyle='--')
ax_hist2.axvline(np.median(feature), color='black', linestyle='-')
histogram_boxplot(churn_df['CreditScore'])
Insights:
histogram_boxplot(churn_df['Age'])
Insights:
histogram_boxplot(churn_df['Tenure'])
Insights:
histogram_boxplot(churn_df['Balance'])
Insights:
histogram_boxplot(churn_df['NumOfProducts'])
Insights:
histogram_boxplot(churn_df['EstimatedSalary'])
Insights:
# select numerical columns
all_col = churn_df.select_dtypes(include=np.number).columns.tolist()
fig, axes = plt.subplots(6, 2, figsize=(20, 15))
fig.suptitle('CDF plot of numerical variables', fontsize=20)
counter = 0
for ii in range(6):
sns.ecdfplot(ax=axes[ii][0],x=churn_df[all_col[counter]])
counter = counter+1
if counter != 11:
sns.ecdfplot(ax=axes[ii][1],x=churn_df[all_col[counter]])
counter = counter+1
else:
pass
fig.tight_layout(pad=2.0)
Insights:
Function to create Bar Charts of percentages
def perc_on_bar(feature):
sns.set(rc={'figure.figsize':(9,3)})
ax=sns.countplot(x=feature, data=churn_df)
total = len(feature)
for p in ax.patches:
percentage = '{:.1f}%'.format(100 * p.get_height()/total)
x = p.get_x() + p.get_width() / 2 - 0.25
y = p.get_y() + p.get_height()
ax.annotate(percentage, (x, y), size = 10)
plt.show()
perc_on_bar(churn_df['Geography'])
Insights:
perc_on_bar(churn_df['Gender'])
Insights:
perc_on_bar(churn_df['NumOfProducts'])
Insights:
perc_on_bar(churn_df['HasCrCard'])
Insights:
perc_on_bar(churn_df['IsActiveMember'])
Insights:
perc_on_bar(churn_df['Exited'])
Insights:
sns.pairplot(churn_df, height=3, diag_kind='kde',hue='Exited');
Insights:
plt.figure(figsize=(10,10))
sns.heatmap(churn_df.corr(), annot=True, fmt=".2")
plt.show()
Insights:
pd.crosstab(churn_df['Geography'],churn_df['Exited'],normalize='index')
| Exited | 0 | 1 |
|---|---|---|
| Geography | ||
| France | 0.838452 | 0.161548 |
| Germany | 0.675568 | 0.324432 |
| Spain | 0.833266 | 0.166734 |
Insights:
pd.crosstab(churn_df['Gender'],churn_df['Exited'],normalize='index')
| Exited | 0 | 1 |
|---|---|---|
| Gender | ||
| Female | 0.749285 | 0.250715 |
| Male | 0.835441 | 0.164559 |
Insights:
pd.crosstab(churn_df['Tenure'],churn_df['Exited'],normalize='index')
| Exited | 0 | 1 |
|---|---|---|
| Tenure | ||
| 0 | 0.769976 | 0.230024 |
| 1 | 0.775845 | 0.224155 |
| 2 | 0.808206 | 0.191794 |
| 3 | 0.788900 | 0.211100 |
| 4 | 0.794742 | 0.205258 |
| 5 | 0.793478 | 0.206522 |
| 6 | 0.797311 | 0.202689 |
| 7 | 0.827821 | 0.172179 |
| 8 | 0.807805 | 0.192195 |
| 9 | 0.783537 | 0.216463 |
| 10 | 0.793878 | 0.206122 |
Insights:
pd.crosstab(churn_df['NumOfProducts'],churn_df['Exited'],normalize='index')
| Exited | 0 | 1 |
|---|---|---|
| NumOfProducts | ||
| 1 | 0.722856 | 0.277144 |
| 2 | 0.924183 | 0.075817 |
| 3 | 0.172932 | 0.827068 |
| 4 | 0.000000 | 1.000000 |
Insights:
pd.crosstab(churn_df['HasCrCard'],churn_df['Exited'],normalize='index')
| Exited | 0 | 1 |
|---|---|---|
| HasCrCard | ||
| 0 | 0.791851 | 0.208149 |
| 1 | 0.798157 | 0.201843 |
Insights:
pd.crosstab(churn_df['IsActiveMember'],churn_df['Exited'],normalize='index')
| Exited | 0 | 1 |
|---|---|---|
| IsActiveMember | ||
| 0 | 0.731491 | 0.268509 |
| 1 | 0.857309 | 0.142691 |
Insights:
#Creating Fuction to plot continous varible distributions
def cont_plot(feature):
sns.distplot(churn_df[churn_df['Exited']==0][feature],color='g',label='Retained Customer', bins=31)
sns.distplot(churn_df[churn_df['Exited']==1][feature],color='r',label='Exited Customer', bins=31)
plt.legend()
plt.show()
cont_plot('CreditScore')
Insights:
cont_plot('Age')
Insights:
cont_plot('Balance')
Insights:
cont_plot('EstimatedSalary')
Insights:
sns.set(rc={"figure.figsize": (10, 7)})
sns.boxplot(y="CreditScore", x="Exited", data=churn_df, orient="vertical");
Insights:
sns.set(rc={"figure.figsize": (10, 7)})
sns.boxplot(y="Age", x="Exited", data=churn_df, orient="vertical");
Insights:
sns.set(rc={"figure.figsize": (10, 7)})
sns.boxplot(y="Tenure", x="Exited", data=churn_df, orient="vertical");
Insights:
sns.set(rc={"figure.figsize": (10, 7)})
sns.boxplot(y="Balance", x="Exited", data=churn_df, orient="vertical");
Insights:
sns.set(rc={"figure.figsize": (10, 7)})
sns.boxplot(y="EstimatedSalary", x="Exited", data=churn_df, orient="vertical");
Insights:
churn_df.groupby(['Exited']).mean()
| RowNumber | CustomerId | CreditScore | Age | Tenure | Balance | NumOfProducts | HasCrCard | IsActiveMember | EstimatedSalary | |
|---|---|---|---|---|---|---|---|---|---|---|
| Exited | ||||||||||
| 0 | 5024.694964 | 1.569117e+07 | 651.853196 | 37.408389 | 5.033279 | 72745.296779 | 1.544267 | 0.707146 | 0.554565 | 99738.391772 |
| 1 | 4905.917526 | 1.569005e+07 | 645.351497 | 44.837997 | 4.932744 | 91108.539337 | 1.475209 | 0.699067 | 0.360825 | 101465.677531 |
Age
sns.set(rc={"figure.figsize": (10, 7)})
sns.boxplot(y="Age", x="Geography", data=churn_df, orient="vertical",hue='Exited');
sns.set(rc={"figure.figsize": (10, 7)})
sns.boxplot(y="Age", x="Gender", data=churn_df, orient="vertical",hue='Exited');
sns.set(rc={"figure.figsize": (10, 7)})
sns.boxplot(y="Age", x="NumOfProducts", data=churn_df, orient="vertical",hue='Exited');
sns.set(rc={"figure.figsize": (10, 7)})
sns.boxplot(y="Age", x="IsActiveMember", data=churn_df, orient="vertical",hue='Exited');
churn_df.pivot_table('Age',index=['Geography','Exited'],aggfunc=['mean','count'])
| mean | count | ||
|---|---|---|---|
| Age | Age | ||
| Geography | Exited | ||
| France | 0 | 37.235966 | 4204 |
| 1 | 45.133333 | 810 | |
| Germany | 0 | 37.311504 | 1695 |
| 1 | 44.894349 | 814 | |
| Spain | 0 | 37.839147 | 2064 |
| 1 | 44.147700 | 413 |
Insights:
Balance
sns.set(rc={"figure.figsize": (10, 7)})
sns.boxplot(y="Balance", x="Geography", data=churn_df, orient="vertical",hue='Exited');
sns.set(rc={"figure.figsize": (10, 7)})
sns.boxplot(y="Balance", x="Gender", data=churn_df, orient="vertical",hue='Exited');
sns.set(rc={"figure.figsize": (10, 7)})
sns.boxplot(y="Balance", x="IsActiveMember", data=churn_df, orient="vertical",hue='Exited');
Insights:
Geography
churn_df.groupby(['Geography']).mean()
| RowNumber | CustomerId | CreditScore | Age | Tenure | Balance | NumOfProducts | HasCrCard | IsActiveMember | EstimatedSalary | Exited | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Geography | |||||||||||
| France | 5025.228560 | 1.569065e+07 | 649.668329 | 38.511767 | 5.004587 | 62092.636516 | 1.530913 | 0.706621 | 0.516753 | 99899.180814 | 0.161548 |
| Germany | 5000.278996 | 1.569056e+07 | 651.453567 | 39.771622 | 5.009964 | 119730.116134 | 1.519729 | 0.713830 | 0.497409 | 101113.435102 | 0.324432 |
| Spain | 4950.667743 | 1.569192e+07 | 651.333872 | 38.890997 | 5.032297 | 61818.147763 | 1.539362 | 0.694792 | 0.529673 | 99440.572281 | 0.166734 |
Insights:
Prepare the data for analysis - Missing value Treatment, Outlier Detection(treat, if needed- why or why not ), Feature Engineering, Prepare data for modeling
Insights:
churn_df.info() #check data again
<class 'pandas.core.frame.DataFrame'> RangeIndex: 10000 entries, 0 to 9999 Data columns (total 14 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 RowNumber 10000 non-null int64 1 CustomerId 10000 non-null int64 2 Surname 10000 non-null category 3 CreditScore 10000 non-null int64 4 Geography 10000 non-null category 5 Gender 10000 non-null category 6 Age 10000 non-null int64 7 Tenure 10000 non-null int64 8 Balance 10000 non-null float64 9 NumOfProducts 10000 non-null int64 10 HasCrCard 10000 non-null int64 11 IsActiveMember 10000 non-null int64 12 EstimatedSalary 10000 non-null float64 13 Exited 10000 non-null int64 dtypes: category(3), float64(2), int64(9) memory usage: 1001.7 KB
data=churn_df.copy() #make copy of data before manipulations
Create Maps to the bins for different variables
#Map for Gender to make it numeric
gender_map={'Male':1,
'Female':0}
Remap Variables
data['Gender'].replace(gender_map,inplace=True)
data.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 10000 entries, 0 to 9999 Data columns (total 14 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 RowNumber 10000 non-null int64 1 CustomerId 10000 non-null int64 2 Surname 10000 non-null category 3 CreditScore 10000 non-null int64 4 Geography 10000 non-null category 5 Gender 10000 non-null int64 6 Age 10000 non-null int64 7 Tenure 10000 non-null int64 8 Balance 10000 non-null float64 9 NumOfProducts 10000 non-null int64 10 HasCrCard 10000 non-null int64 11 IsActiveMember 10000 non-null int64 12 EstimatedSalary 10000 non-null float64 13 Exited 10000 non-null int64 dtypes: category(2), float64(2), int64(10) memory usage: 1.0 MB
One hot encoding for Select category variables
# Create one hot encoding
oneHotCols=["Geography"]
data=pd.get_dummies(data, columns=oneHotCols,drop_first=True)
Check to see all the data types are numeric before model building
data.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 10000 entries, 0 to 9999 Data columns (total 15 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 RowNumber 10000 non-null int64 1 CustomerId 10000 non-null int64 2 Surname 10000 non-null category 3 CreditScore 10000 non-null int64 4 Gender 10000 non-null int64 5 Age 10000 non-null int64 6 Tenure 10000 non-null int64 7 Balance 10000 non-null float64 8 NumOfProducts 10000 non-null int64 9 HasCrCard 10000 non-null int64 10 IsActiveMember 10000 non-null int64 11 EstimatedSalary 10000 non-null float64 12 Exited 10000 non-null int64 13 Geography_Germany 10000 non-null uint8 14 Geography_Spain 10000 non-null uint8 dtypes: category(1), float64(2), int64(10), uint8(2) memory usage: 1.1 MB
Insights:
Drop unneeded variables
data.drop(["RowNumber",'CustomerId','Surname'] , axis=1,inplace=True)
Insights:
# Spliting the target data
X_data = data.drop(columns=['Exited'])
y_data = data['Exited']
#printing the shape of the data
print(y_data.shape)
print(X_data.shape)
(10000,) (10000, 11)
X_data.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 10000 entries, 0 to 9999 Data columns (total 11 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 CreditScore 10000 non-null int64 1 Gender 10000 non-null int64 2 Age 10000 non-null int64 3 Tenure 10000 non-null int64 4 Balance 10000 non-null float64 5 NumOfProducts 10000 non-null int64 6 HasCrCard 10000 non-null int64 7 IsActiveMember 10000 non-null int64 8 EstimatedSalary 10000 non-null float64 9 Geography_Germany 10000 non-null uint8 10 Geography_Spain 10000 non-null uint8 dtypes: float64(2), int64(7), uint8(2) memory usage: 722.8 KB
Insights:
# Scaling the data set before modeling
scaler=StandardScaler()
subset=X_data.copy()
subset_scaled=scaler.fit_transform(subset)
subset_scaled
subset_scaled_df=pd.DataFrame(subset_scaled,columns=subset.columns)
subset_scaled_df
| CreditScore | Gender | Age | Tenure | Balance | NumOfProducts | HasCrCard | IsActiveMember | EstimatedSalary | Geography_Germany | Geography_Spain | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | -0.326221 | -1.095988 | 0.293517 | -1.041760 | -1.225848 | -0.911583 | 0.646092 | 0.970243 | 0.021886 | -0.578736 | -0.573809 |
| 1 | -0.440036 | -1.095988 | 0.198164 | -1.387538 | 0.117350 | -0.911583 | -1.547768 | 0.970243 | 0.216534 | -0.578736 | 1.742740 |
| 2 | -1.536794 | -1.095988 | 0.293517 | 1.032908 | 1.333053 | 2.527057 | 0.646092 | -1.030670 | 0.240687 | -0.578736 | -0.573809 |
| 3 | 0.501521 | -1.095988 | 0.007457 | -1.387538 | -1.225848 | 0.807737 | -1.547768 | -1.030670 | -0.108918 | -0.578736 | -0.573809 |
| 4 | 2.063884 | -1.095988 | 0.388871 | -1.041760 | 0.785728 | -0.911583 | 0.646092 | 0.970243 | -0.365276 | -0.578736 | 1.742740 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 9995 | 1.246488 | 0.912419 | 0.007457 | -0.004426 | -1.225848 | 0.807737 | 0.646092 | -1.030670 | -0.066419 | -0.578736 | -0.573809 |
| 9996 | -1.391939 | 0.912419 | -0.373958 | 1.724464 | -0.306379 | -0.911583 | 0.646092 | 0.970243 | 0.027988 | -0.578736 | -0.573809 |
| 9997 | 0.604988 | -1.095988 | -0.278604 | 0.687130 | -1.225848 | -0.911583 | -1.547768 | 0.970243 | -1.008643 | -0.578736 | -0.573809 |
| 9998 | 1.256835 | 0.912419 | 0.293517 | -0.695982 | -0.022608 | 0.807737 | 0.646092 | -1.030670 | -0.125231 | 1.727904 | -0.573809 |
| 9999 | 1.463771 | -1.095988 | -1.041433 | -0.350204 | 0.859965 | -0.911583 | 0.646092 | -1.030670 | -1.076370 | -0.578736 | -0.573809 |
10000 rows × 11 columns
subset_scaled_df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 10000 entries, 0 to 9999 Data columns (total 11 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 CreditScore 10000 non-null float64 1 Gender 10000 non-null float64 2 Age 10000 non-null float64 3 Tenure 10000 non-null float64 4 Balance 10000 non-null float64 5 NumOfProducts 10000 non-null float64 6 HasCrCard 10000 non-null float64 7 IsActiveMember 10000 non-null float64 8 EstimatedSalary 10000 non-null float64 9 Geography_Germany 10000 non-null float64 10 Geography_Spain 10000 non-null float64 dtypes: float64(11) memory usage: 859.5 KB
X_train, X_test, y_train, y_test = train_test_split(subset_scaled_df, y_data, test_size = 0.2, random_state = 8)
print(X_train.shape)
print(X_test.shape)
print(y_train.shape)
print(y_test.shape)
(8000, 11) (2000, 11) (8000,) (2000,)
Model 1: basic DNN Model
For Model will be a basic model with 1 hidden layer with 8 nodes. Also, since this is a binary class the output will be of class 1
#initialize the model
model = Sequential()
# Creating input layer and first hiden layer
model.add(Dense(units=8, input_dim = 11,activation='relu')) # input of 11 columns as shown above
# hidden layer
model.add(Dense(1,activation='sigmoid')) # binary classification for Churn or not
# Create optimizer with default learning rate
# Compile the model
model.compile(optimizer='adam',loss='binary_crossentropy',metrics=['accuracy'])
#create model summary
model.summary()
Model: "sequential_2" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= dense_5 (Dense) (None, 8) 96 _________________________________________________________________ dense_6 (Dense) (None, 1) 9 ================================================================= Total params: 105 Trainable params: 105 Non-trainable params: 0 _________________________________________________________________
Training [Forward pass and Backpropagation]
#fitting the model
history=model.fit(X_train,y_train,batch_size=20,epochs=100,validation_split=0.2 )
Epoch 1/100 320/320 [==============================] - 2s 3ms/step - loss: 0.7461 - accuracy: 0.5341 - val_loss: 0.5356 - val_accuracy: 0.7688 Epoch 2/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4866 - accuracy: 0.8055 - val_loss: 0.4921 - val_accuracy: 0.7806 Epoch 3/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4386 - accuracy: 0.8171 - val_loss: 0.4779 - val_accuracy: 0.7856 Epoch 4/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4283 - accuracy: 0.8187 - val_loss: 0.4706 - val_accuracy: 0.7956 Epoch 5/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4440 - accuracy: 0.8025 - val_loss: 0.4636 - val_accuracy: 0.7931 Epoch 6/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4082 - accuracy: 0.8264 - val_loss: 0.4530 - val_accuracy: 0.8006 Epoch 7/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4168 - accuracy: 0.8256 - val_loss: 0.4413 - val_accuracy: 0.8044 Epoch 8/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3908 - accuracy: 0.8278 - val_loss: 0.4284 - val_accuracy: 0.8119 Epoch 9/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3757 - accuracy: 0.8406 - val_loss: 0.4149 - val_accuracy: 0.8225 Epoch 10/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3821 - accuracy: 0.8390 - val_loss: 0.4052 - val_accuracy: 0.8238 Epoch 11/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3604 - accuracy: 0.8467 - val_loss: 0.3964 - val_accuracy: 0.8363 Epoch 12/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3609 - accuracy: 0.8472 - val_loss: 0.3904 - val_accuracy: 0.8375 Epoch 13/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3476 - accuracy: 0.8569 - val_loss: 0.3846 - val_accuracy: 0.8381 Epoch 14/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3396 - accuracy: 0.8620 - val_loss: 0.3812 - val_accuracy: 0.8388 Epoch 15/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3516 - accuracy: 0.8556 - val_loss: 0.3811 - val_accuracy: 0.8419 Epoch 16/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3407 - accuracy: 0.8585 - val_loss: 0.3755 - val_accuracy: 0.8400 Epoch 17/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3528 - accuracy: 0.8535 - val_loss: 0.3759 - val_accuracy: 0.8425 Epoch 18/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3351 - accuracy: 0.8582 - val_loss: 0.3746 - val_accuracy: 0.8425 Epoch 19/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3440 - accuracy: 0.8618 - val_loss: 0.3741 - val_accuracy: 0.8431 Epoch 20/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3315 - accuracy: 0.8632 - val_loss: 0.3716 - val_accuracy: 0.8406 Epoch 21/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3279 - accuracy: 0.8627 - val_loss: 0.3711 - val_accuracy: 0.8431 Epoch 22/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3404 - accuracy: 0.8611 - val_loss: 0.3711 - val_accuracy: 0.8450 Epoch 23/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3416 - accuracy: 0.8593 - val_loss: 0.3709 - val_accuracy: 0.8438 Epoch 24/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3345 - accuracy: 0.8611 - val_loss: 0.3705 - val_accuracy: 0.8450 Epoch 25/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3198 - accuracy: 0.8678 - val_loss: 0.3691 - val_accuracy: 0.8462 Epoch 26/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3299 - accuracy: 0.8581 - val_loss: 0.3691 - val_accuracy: 0.8462 Epoch 27/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3299 - accuracy: 0.8654 - val_loss: 0.3695 - val_accuracy: 0.8475 Epoch 28/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3335 - accuracy: 0.8647 - val_loss: 0.3701 - val_accuracy: 0.8500 Epoch 29/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3317 - accuracy: 0.8627 - val_loss: 0.3690 - val_accuracy: 0.8481 Epoch 30/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3319 - accuracy: 0.8620 - val_loss: 0.3679 - val_accuracy: 0.8469 Epoch 31/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3320 - accuracy: 0.8626 - val_loss: 0.3684 - val_accuracy: 0.8487 Epoch 32/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3342 - accuracy: 0.8634 - val_loss: 0.3679 - val_accuracy: 0.8481 Epoch 33/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3289 - accuracy: 0.8653 - val_loss: 0.3698 - val_accuracy: 0.8494 Epoch 34/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3256 - accuracy: 0.8660 - val_loss: 0.3689 - val_accuracy: 0.8500 Epoch 35/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3257 - accuracy: 0.8649 - val_loss: 0.3670 - val_accuracy: 0.8481 Epoch 36/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3302 - accuracy: 0.8613 - val_loss: 0.3677 - val_accuracy: 0.8500 Epoch 37/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3230 - accuracy: 0.8708 - val_loss: 0.3686 - val_accuracy: 0.8506 Epoch 38/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3259 - accuracy: 0.8652 - val_loss: 0.3657 - val_accuracy: 0.8475 Epoch 39/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3368 - accuracy: 0.8570 - val_loss: 0.3687 - val_accuracy: 0.8469 Epoch 40/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3345 - accuracy: 0.8614 - val_loss: 0.3685 - val_accuracy: 0.8500 Epoch 41/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3366 - accuracy: 0.8658 - val_loss: 0.3677 - val_accuracy: 0.8494 Epoch 42/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3390 - accuracy: 0.8628 - val_loss: 0.3677 - val_accuracy: 0.8506 Epoch 43/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3280 - accuracy: 0.8610 - val_loss: 0.3683 - val_accuracy: 0.8494 Epoch 44/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3326 - accuracy: 0.8596 - val_loss: 0.3664 - val_accuracy: 0.8500 Epoch 45/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3215 - accuracy: 0.8670 - val_loss: 0.3673 - val_accuracy: 0.8481 Epoch 46/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3343 - accuracy: 0.8619 - val_loss: 0.3673 - val_accuracy: 0.8506 Epoch 47/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3325 - accuracy: 0.8586 - val_loss: 0.3663 - val_accuracy: 0.8500 Epoch 48/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3253 - accuracy: 0.8699 - val_loss: 0.3676 - val_accuracy: 0.8494 Epoch 49/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3277 - accuracy: 0.8619 - val_loss: 0.3650 - val_accuracy: 0.8487 Epoch 50/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3310 - accuracy: 0.8644 - val_loss: 0.3665 - val_accuracy: 0.8469 Epoch 51/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3350 - accuracy: 0.8577 - val_loss: 0.3649 - val_accuracy: 0.8481 Epoch 52/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3387 - accuracy: 0.8600 - val_loss: 0.3655 - val_accuracy: 0.8481 Epoch 53/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3292 - accuracy: 0.8604 - val_loss: 0.3667 - val_accuracy: 0.8481 Epoch 54/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3442 - accuracy: 0.8606 - val_loss: 0.3661 - val_accuracy: 0.8475 Epoch 55/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3284 - accuracy: 0.8623 - val_loss: 0.3650 - val_accuracy: 0.8500 Epoch 56/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3258 - accuracy: 0.8691 - val_loss: 0.3640 - val_accuracy: 0.8481 Epoch 57/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3281 - accuracy: 0.8636 - val_loss: 0.3643 - val_accuracy: 0.8450 Epoch 58/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3329 - accuracy: 0.8647 - val_loss: 0.3660 - val_accuracy: 0.8512 Epoch 59/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3265 - accuracy: 0.8655 - val_loss: 0.3649 - val_accuracy: 0.8506 Epoch 60/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3282 - accuracy: 0.8668 - val_loss: 0.3651 - val_accuracy: 0.8494 Epoch 61/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3283 - accuracy: 0.8680 - val_loss: 0.3652 - val_accuracy: 0.8475 Epoch 62/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3347 - accuracy: 0.8606 - val_loss: 0.3647 - val_accuracy: 0.8462 Epoch 63/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3463 - accuracy: 0.8518 - val_loss: 0.3659 - val_accuracy: 0.8494 Epoch 64/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3301 - accuracy: 0.8657 - val_loss: 0.3640 - val_accuracy: 0.8462 Epoch 65/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3236 - accuracy: 0.8665 - val_loss: 0.3652 - val_accuracy: 0.8469 Epoch 66/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3293 - accuracy: 0.8646 - val_loss: 0.3651 - val_accuracy: 0.8500 Epoch 67/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3236 - accuracy: 0.8653 - val_loss: 0.3640 - val_accuracy: 0.8494 Epoch 68/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3196 - accuracy: 0.8703 - val_loss: 0.3638 - val_accuracy: 0.8456 Epoch 69/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3243 - accuracy: 0.8679 - val_loss: 0.3647 - val_accuracy: 0.8519 Epoch 70/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3240 - accuracy: 0.8660 - val_loss: 0.3639 - val_accuracy: 0.8475 Epoch 71/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3322 - accuracy: 0.8613 - val_loss: 0.3632 - val_accuracy: 0.8425 Epoch 72/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3290 - accuracy: 0.8611 - val_loss: 0.3670 - val_accuracy: 0.8512 Epoch 73/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3319 - accuracy: 0.8589 - val_loss: 0.3660 - val_accuracy: 0.8506 Epoch 74/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3170 - accuracy: 0.8693 - val_loss: 0.3627 - val_accuracy: 0.8456 Epoch 75/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3261 - accuracy: 0.8645 - val_loss: 0.3629 - val_accuracy: 0.8475 Epoch 76/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3293 - accuracy: 0.8686 - val_loss: 0.3637 - val_accuracy: 0.8500 Epoch 77/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3309 - accuracy: 0.8612 - val_loss: 0.3648 - val_accuracy: 0.8481 Epoch 78/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3200 - accuracy: 0.8655 - val_loss: 0.3632 - val_accuracy: 0.8450 Epoch 79/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3252 - accuracy: 0.8609 - val_loss: 0.3647 - val_accuracy: 0.8481 Epoch 80/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3256 - accuracy: 0.8652 - val_loss: 0.3639 - val_accuracy: 0.8450 Epoch 81/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3236 - accuracy: 0.8651 - val_loss: 0.3647 - val_accuracy: 0.8487 Epoch 82/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3465 - accuracy: 0.8571 - val_loss: 0.3656 - val_accuracy: 0.8494 Epoch 83/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3199 - accuracy: 0.8708 - val_loss: 0.3646 - val_accuracy: 0.8481 Epoch 84/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3200 - accuracy: 0.8684 - val_loss: 0.3620 - val_accuracy: 0.8462 Epoch 85/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3166 - accuracy: 0.8702 - val_loss: 0.3638 - val_accuracy: 0.8475 Epoch 86/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3245 - accuracy: 0.8646 - val_loss: 0.3639 - val_accuracy: 0.8456 Epoch 87/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3270 - accuracy: 0.8625 - val_loss: 0.3619 - val_accuracy: 0.8438 Epoch 88/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3384 - accuracy: 0.8532 - val_loss: 0.3658 - val_accuracy: 0.8525 Epoch 89/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3141 - accuracy: 0.8692 - val_loss: 0.3636 - val_accuracy: 0.8469 Epoch 90/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3270 - accuracy: 0.8598 - val_loss: 0.3650 - val_accuracy: 0.8500 Epoch 91/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3298 - accuracy: 0.8645 - val_loss: 0.3636 - val_accuracy: 0.8475 Epoch 92/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3201 - accuracy: 0.8634 - val_loss: 0.3636 - val_accuracy: 0.8450 Epoch 93/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3426 - accuracy: 0.8542 - val_loss: 0.3647 - val_accuracy: 0.8487 Epoch 94/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3219 - accuracy: 0.8668 - val_loss: 0.3632 - val_accuracy: 0.8462 Epoch 95/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3277 - accuracy: 0.8655 - val_loss: 0.3638 - val_accuracy: 0.8475 Epoch 96/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3199 - accuracy: 0.8708 - val_loss: 0.3655 - val_accuracy: 0.8500 Epoch 97/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3291 - accuracy: 0.8647 - val_loss: 0.3635 - val_accuracy: 0.8469 Epoch 98/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3259 - accuracy: 0.8625 - val_loss: 0.3658 - val_accuracy: 0.8512 Epoch 99/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3267 - accuracy: 0.8594 - val_loss: 0.3637 - val_accuracy: 0.8481 Epoch 100/100 320/320 [==============================] - 1s 2ms/step - loss: 0.3138 - accuracy: 0.8695 - val_loss: 0.3629 - val_accuracy: 0.8475
Plotting the train and test loss
# Capturing learning history per epoch
hist = pd.DataFrame(history.history)
hist['epoch'] = history.epoch
# Plotting accuracy at different epochs
plt.plot(hist['loss'])
plt.plot(hist['val_loss'])
plt.legend(("train" , "valid") , loc =0)
<matplotlib.legend.Legend at 0x7fe3cf0e1c50>
Lets evalute the 1st model against the test data
score = model.evaluate(X_test, y_test)
63/63 [==============================] - 0s 2ms/step - loss: 0.3394 - accuracy: 0.8630
print(1-sum(y_test)/len(y_test),"would have been accuracy if we guessed all 0s")
0.7935 would have been accuracy if we guessed all 0s
The model did better than if we would have guessed zero on all of the for all the predictions.Also,there doesn't appear to be any severe overfitting
def make_confusion_matrix(cf,
group_names=None,
categories='auto',
count=True,
percent=True,
cbar=True,
xyticks=True,
xyplotlabels=True,
sum_stats=True,
figsize=None,
cmap='Blues',
title=None):
'''
This function will make a pretty plot of an sklearn Confusion Matrix cm using a Seaborn heatmap visualization.
Arguments
'''
# CODE TO GENERATE TEXT INSIDE EACH SQUARE
blanks = ['' for i in range(cf.size)]
if group_names and len(group_names)==cf.size:
group_labels = ["{}\n".format(value) for value in group_names]
else:
group_labels = blanks
if count:
group_counts = ["{0:0.0f}\n".format(value) for value in cf.flatten()]
else:
group_counts = blanks
if percent:
group_percentages = ["{0:.2%}".format(value) for value in cf.flatten()/np.sum(cf)]
else:
group_percentages = blanks
box_labels = [f"{v1}{v2}{v3}".strip() for v1, v2, v3 in zip(group_labels,group_counts,group_percentages)]
box_labels = np.asarray(box_labels).reshape(cf.shape[0],cf.shape[1])
# CODE TO GENERATE SUMMARY STATISTICS & TEXT FOR SUMMARY STATS
if sum_stats:
#Accuracy is sum of diagonal divided by total observations
accuracy = np.trace(cf) / float(np.sum(cf))
#if it is a binary confusion matrix, show some more stats
if len(cf)==2:
#Metrics for Binary Confusion Matrices
precision = cf[1,1] / sum(cf[:,1])
recall = cf[1,1] / sum(cf[1,:])
f1_score = 2*precision*recall / (precision + recall)
stats_text = "\n\nAccuracy={:0.3f}\nPrecision={:0.3f}\nRecall={:0.3f}\nF1 Score={:0.3f}".format(
accuracy,precision,recall,f1_score)
else:
stats_text = "\n\nAccuracy={:0.3f}".format(accuracy)
else:
stats_text = ""
# SET FIGURE PARAMETERS ACCORDING TO OTHER ARGUMENTS
if figsize==None:
#Get default figure size if not set
figsize = plt.rcParams.get('figure.figsize')
if xyticks==False:
#Do not show categories if xyticks is False
categories=False
# MAKE THE HEATMAP VISUALIZATION
plt.figure(figsize=figsize)
sns.heatmap(cf,annot=box_labels,fmt="",cmap=cmap,cbar=cbar,xticklabels=categories,yticklabels=categories)
if xyplotlabels:
plt.ylabel('True label')
plt.xlabel('Predicted label' + stats_text)
else:
plt.xlabel(stats_text)
if title:
plt.title(title)
## Confusion Matrix
import seaborn as sn
y_pred1 = model.predict(X_test)
for i in range(len(y_test)):
if y_pred1[i]>0.5:
y_pred1[i]=1
else:
y_pred1[i]=0
cm2=confusion_matrix(y_test, y_pred1)
labels = ['True Negative','False Positive','False Negative','True Positive']
categories = [ 'Retained','Exited']
make_confusion_matrix(cm2,
group_names=labels,
categories=categories,
cmap='Blues')
Insights:
#AUC ROC curve
from sklearn.metrics import roc_auc_score
from sklearn.metrics import roc_curve
DNN_roc_auc = roc_auc_score(y_test, model.predict(X_test))
fpr, tpr, thresholds = roc_curve(y_test, model.predict(X_test))
# calculate the g-mean for each threshold
gmeans = np.sqrt(tpr * (1-fpr))
# locate the index of the largest g-mean
ix = np.argmax(gmeans)
print('Best Threshold=%f, G-Mean=%.3f' % (thresholds[ix], gmeans[ix]))
plt.figure()
plt.scatter(fpr[ix], tpr[ix], marker='o', color='black', label='Best')
plt.plot(fpr, tpr, label='Model 1 (area = %0.2f)' % DNN_roc_auc)
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic')
plt.legend(loc="lower right")
plt.savefig('ROC')
plt.show()
Best Threshold=0.231908, G-Mean=0.797
Insights:
y_pred1 = model.predict(X_test)
for i in range(len(y_test)):
if y_pred1[i]>0.231908:
y_pred1[i]=1
else:
y_pred1[i]=0
cm2=confusion_matrix(y_test, y_pred1)
labels = ['True Negative','False Positive','False Negative','True Positive']
categories = [ 'Retained','Exited']
make_confusion_matrix(cm2,
group_names=labels,
categories=categories,
cmap='Blues')
Insights:
Model2
from sklearn.utils import class_weight
class_weights = class_weight.compute_class_weight('balanced', np.unique(y_train), np.array([y_train.iloc[i] for i in range(len(y_train))]))
class_weights = dict(enumerate(class_weights))
class_weights
{0: 0.6273525721455459, 1: 2.4630541871921183}
#initialize the model
model2 = Sequential()
# Creating input layer and first hiden layer
model2.add(Dense(units=8, input_dim = 11,activation='relu',kernel_initializer='he_normal')) # input of 11 columns as shown above
model2.add(Dense(units=8,activation='relu',kernel_initializer='he_normal'))
# hidden layer
model2.add(Dense(1,activation='sigmoid')) # binary classification for Churn or not
# Create optimizer with default learning rate
# Compile the model
model2.compile(Adam(lr=0.001),loss='binary_crossentropy',metrics=['accuracy'])
#create model summary
model2.summary()
Model: "sequential_3" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= dense_7 (Dense) (None, 8) 96 _________________________________________________________________ dense_8 (Dense) (None, 8) 72 _________________________________________________________________ dense_9 (Dense) (None, 1) 9 ================================================================= Total params: 177 Trainable params: 177 Non-trainable params: 0 _________________________________________________________________
Training [Forward pass and Backpropagation]
#fitting the model
history=model2.fit(X_train,y_train,batch_size=20,epochs=100,validation_split=0.2 ,class_weight=class_weights)
Epoch 1/100 320/320 [==============================] - 2s 3ms/step - loss: 0.8527 - accuracy: 0.6723 - val_loss: 0.6616 - val_accuracy: 0.6206 Epoch 2/100 320/320 [==============================] - 1s 2ms/step - loss: 0.6610 - accuracy: 0.6124 - val_loss: 0.6318 - val_accuracy: 0.6269 Epoch 3/100 320/320 [==============================] - 1s 2ms/step - loss: 0.6260 - accuracy: 0.6325 - val_loss: 0.6154 - val_accuracy: 0.6356 Epoch 4/100 320/320 [==============================] - 1s 2ms/step - loss: 0.6047 - accuracy: 0.6619 - val_loss: 0.6067 - val_accuracy: 0.6594 Epoch 5/100 320/320 [==============================] - 1s 2ms/step - loss: 0.5957 - accuracy: 0.6685 - val_loss: 0.5756 - val_accuracy: 0.6931 Epoch 6/100 320/320 [==============================] - 1s 2ms/step - loss: 0.5819 - accuracy: 0.6962 - val_loss: 0.5757 - val_accuracy: 0.6956 Epoch 7/100 320/320 [==============================] - 1s 2ms/step - loss: 0.5600 - accuracy: 0.7013 - val_loss: 0.5769 - val_accuracy: 0.7044 Epoch 8/100 320/320 [==============================] - 1s 2ms/step - loss: 0.5803 - accuracy: 0.6910 - val_loss: 0.5716 - val_accuracy: 0.7100 Epoch 9/100 320/320 [==============================] - 1s 2ms/step - loss: 0.5667 - accuracy: 0.7028 - val_loss: 0.5665 - val_accuracy: 0.7169 Epoch 10/100 320/320 [==============================] - 1s 2ms/step - loss: 0.5611 - accuracy: 0.7136 - val_loss: 0.5535 - val_accuracy: 0.7237 Epoch 11/100 320/320 [==============================] - 1s 2ms/step - loss: 0.5592 - accuracy: 0.7163 - val_loss: 0.5453 - val_accuracy: 0.7275 Epoch 12/100 320/320 [==============================] - 1s 2ms/step - loss: 0.5472 - accuracy: 0.7417 - val_loss: 0.5515 - val_accuracy: 0.7287 Epoch 13/100 320/320 [==============================] - 1s 2ms/step - loss: 0.5519 - accuracy: 0.7288 - val_loss: 0.5324 - val_accuracy: 0.7356 Epoch 14/100 320/320 [==============================] - 1s 2ms/step - loss: 0.5411 - accuracy: 0.7473 - val_loss: 0.5365 - val_accuracy: 0.7331 Epoch 15/100 320/320 [==============================] - 1s 2ms/step - loss: 0.5396 - accuracy: 0.7291 - val_loss: 0.5395 - val_accuracy: 0.7337 Epoch 16/100 320/320 [==============================] - 1s 2ms/step - loss: 0.5352 - accuracy: 0.7365 - val_loss: 0.5301 - val_accuracy: 0.7337 Epoch 17/100 320/320 [==============================] - 1s 2ms/step - loss: 0.5302 - accuracy: 0.7495 - val_loss: 0.5266 - val_accuracy: 0.7356 Epoch 18/100 320/320 [==============================] - 1s 2ms/step - loss: 0.5229 - accuracy: 0.7513 - val_loss: 0.5448 - val_accuracy: 0.7256 Epoch 19/100 320/320 [==============================] - 1s 2ms/step - loss: 0.5187 - accuracy: 0.7455 - val_loss: 0.5323 - val_accuracy: 0.7375 Epoch 20/100 320/320 [==============================] - 1s 2ms/step - loss: 0.5150 - accuracy: 0.7523 - val_loss: 0.5361 - val_accuracy: 0.7256 Epoch 21/100 320/320 [==============================] - 1s 2ms/step - loss: 0.5231 - accuracy: 0.7479 - val_loss: 0.5072 - val_accuracy: 0.7500 Epoch 22/100 320/320 [==============================] - 1s 2ms/step - loss: 0.5114 - accuracy: 0.7610 - val_loss: 0.5156 - val_accuracy: 0.7437 Epoch 23/100 320/320 [==============================] - 1s 2ms/step - loss: 0.5131 - accuracy: 0.7537 - val_loss: 0.5016 - val_accuracy: 0.7569 Epoch 24/100 320/320 [==============================] - 1s 2ms/step - loss: 0.5243 - accuracy: 0.7558 - val_loss: 0.4905 - val_accuracy: 0.7656 Epoch 25/100 320/320 [==============================] - 1s 2ms/step - loss: 0.5086 - accuracy: 0.7646 - val_loss: 0.5157 - val_accuracy: 0.7425 Epoch 26/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4863 - accuracy: 0.7695 - val_loss: 0.5124 - val_accuracy: 0.7462 Epoch 27/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4915 - accuracy: 0.7597 - val_loss: 0.5054 - val_accuracy: 0.7481 Epoch 28/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4939 - accuracy: 0.7648 - val_loss: 0.5068 - val_accuracy: 0.7513 Epoch 29/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4899 - accuracy: 0.7641 - val_loss: 0.4905 - val_accuracy: 0.7619 Epoch 30/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4861 - accuracy: 0.7738 - val_loss: 0.5001 - val_accuracy: 0.7588 Epoch 31/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4820 - accuracy: 0.7716 - val_loss: 0.4861 - val_accuracy: 0.7681 Epoch 32/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4673 - accuracy: 0.7854 - val_loss: 0.4889 - val_accuracy: 0.7675 Epoch 33/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4909 - accuracy: 0.7803 - val_loss: 0.4899 - val_accuracy: 0.7700 Epoch 34/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4655 - accuracy: 0.7798 - val_loss: 0.4951 - val_accuracy: 0.7631 Epoch 35/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4625 - accuracy: 0.7908 - val_loss: 0.4903 - val_accuracy: 0.7694 Epoch 36/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4646 - accuracy: 0.7836 - val_loss: 0.4693 - val_accuracy: 0.7812 Epoch 37/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4722 - accuracy: 0.7873 - val_loss: 0.4699 - val_accuracy: 0.7819 Epoch 38/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4735 - accuracy: 0.7979 - val_loss: 0.4769 - val_accuracy: 0.7775 Epoch 39/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4415 - accuracy: 0.8018 - val_loss: 0.4815 - val_accuracy: 0.7756 Epoch 40/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4710 - accuracy: 0.7851 - val_loss: 0.4873 - val_accuracy: 0.7688 Epoch 41/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4477 - accuracy: 0.7892 - val_loss: 0.4678 - val_accuracy: 0.7825 Epoch 42/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4611 - accuracy: 0.7858 - val_loss: 0.4599 - val_accuracy: 0.7900 Epoch 43/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4601 - accuracy: 0.7962 - val_loss: 0.4690 - val_accuracy: 0.7825 Epoch 44/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4688 - accuracy: 0.7843 - val_loss: 0.4621 - val_accuracy: 0.7831 Epoch 45/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4694 - accuracy: 0.7887 - val_loss: 0.4584 - val_accuracy: 0.7869 Epoch 46/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4656 - accuracy: 0.7872 - val_loss: 0.4942 - val_accuracy: 0.7631 Epoch 47/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4467 - accuracy: 0.7938 - val_loss: 0.4805 - val_accuracy: 0.7788 Epoch 48/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4631 - accuracy: 0.7933 - val_loss: 0.4695 - val_accuracy: 0.7856 Epoch 49/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4656 - accuracy: 0.7886 - val_loss: 0.4778 - val_accuracy: 0.7750 Epoch 50/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4581 - accuracy: 0.7911 - val_loss: 0.4779 - val_accuracy: 0.7738 Epoch 51/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4514 - accuracy: 0.7873 - val_loss: 0.4774 - val_accuracy: 0.7713 Epoch 52/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4641 - accuracy: 0.7819 - val_loss: 0.4667 - val_accuracy: 0.7825 Epoch 53/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4590 - accuracy: 0.7915 - val_loss: 0.4735 - val_accuracy: 0.7756 Epoch 54/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4521 - accuracy: 0.7848 - val_loss: 0.4453 - val_accuracy: 0.7981 Epoch 55/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4422 - accuracy: 0.8063 - val_loss: 0.4732 - val_accuracy: 0.7806 Epoch 56/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4522 - accuracy: 0.7977 - val_loss: 0.4710 - val_accuracy: 0.7788 Epoch 57/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4766 - accuracy: 0.7838 - val_loss: 0.4642 - val_accuracy: 0.7850 Epoch 58/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4457 - accuracy: 0.7964 - val_loss: 0.4929 - val_accuracy: 0.7644 Epoch 59/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4578 - accuracy: 0.7827 - val_loss: 0.4856 - val_accuracy: 0.7681 Epoch 60/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4535 - accuracy: 0.7938 - val_loss: 0.4525 - val_accuracy: 0.7937 Epoch 61/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4501 - accuracy: 0.8071 - val_loss: 0.4711 - val_accuracy: 0.7831 Epoch 62/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4552 - accuracy: 0.7923 - val_loss: 0.4704 - val_accuracy: 0.7788 Epoch 63/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4559 - accuracy: 0.7950 - val_loss: 0.4709 - val_accuracy: 0.7794 Epoch 64/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4530 - accuracy: 0.7904 - val_loss: 0.4712 - val_accuracy: 0.7769 Epoch 65/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4344 - accuracy: 0.8049 - val_loss: 0.4841 - val_accuracy: 0.7681 Epoch 66/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4398 - accuracy: 0.8016 - val_loss: 0.4663 - val_accuracy: 0.7812 Epoch 67/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4528 - accuracy: 0.7953 - val_loss: 0.4711 - val_accuracy: 0.7788 Epoch 68/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4443 - accuracy: 0.7993 - val_loss: 0.4721 - val_accuracy: 0.7763 Epoch 69/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4311 - accuracy: 0.8040 - val_loss: 0.4695 - val_accuracy: 0.7775 Epoch 70/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4494 - accuracy: 0.7966 - val_loss: 0.4496 - val_accuracy: 0.7906 Epoch 71/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4529 - accuracy: 0.7958 - val_loss: 0.4480 - val_accuracy: 0.7912 Epoch 72/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4519 - accuracy: 0.7993 - val_loss: 0.4351 - val_accuracy: 0.8025 Epoch 73/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4387 - accuracy: 0.8096 - val_loss: 0.4706 - val_accuracy: 0.7788 Epoch 74/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4422 - accuracy: 0.8010 - val_loss: 0.4572 - val_accuracy: 0.7837 Epoch 75/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4336 - accuracy: 0.8097 - val_loss: 0.5044 - val_accuracy: 0.7487 Epoch 76/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4406 - accuracy: 0.7860 - val_loss: 0.4667 - val_accuracy: 0.7756 Epoch 77/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4529 - accuracy: 0.7958 - val_loss: 0.4697 - val_accuracy: 0.7744 Epoch 78/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4350 - accuracy: 0.7985 - val_loss: 0.4524 - val_accuracy: 0.7862 Epoch 79/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4460 - accuracy: 0.8060 - val_loss: 0.4601 - val_accuracy: 0.7781 Epoch 80/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4427 - accuracy: 0.8014 - val_loss: 0.4653 - val_accuracy: 0.7800 Epoch 81/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4382 - accuracy: 0.8059 - val_loss: 0.4633 - val_accuracy: 0.7788 Epoch 82/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4277 - accuracy: 0.8089 - val_loss: 0.4672 - val_accuracy: 0.7788 Epoch 83/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4339 - accuracy: 0.8068 - val_loss: 0.4857 - val_accuracy: 0.7713 Epoch 84/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4465 - accuracy: 0.7967 - val_loss: 0.4712 - val_accuracy: 0.7725 Epoch 85/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4509 - accuracy: 0.7960 - val_loss: 0.4600 - val_accuracy: 0.7812 Epoch 86/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4459 - accuracy: 0.7989 - val_loss: 0.4548 - val_accuracy: 0.7912 Epoch 87/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4451 - accuracy: 0.8058 - val_loss: 0.4609 - val_accuracy: 0.7812 Epoch 88/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4487 - accuracy: 0.7940 - val_loss: 0.4740 - val_accuracy: 0.7706 Epoch 89/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4370 - accuracy: 0.8015 - val_loss: 0.4614 - val_accuracy: 0.7819 Epoch 90/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4548 - accuracy: 0.7969 - val_loss: 0.4794 - val_accuracy: 0.7713 Epoch 91/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4361 - accuracy: 0.8035 - val_loss: 0.4937 - val_accuracy: 0.7588 Epoch 92/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4389 - accuracy: 0.7959 - val_loss: 0.4675 - val_accuracy: 0.7781 Epoch 93/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4448 - accuracy: 0.7981 - val_loss: 0.4637 - val_accuracy: 0.7819 Epoch 94/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4394 - accuracy: 0.8061 - val_loss: 0.4652 - val_accuracy: 0.7788 Epoch 95/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4399 - accuracy: 0.7961 - val_loss: 0.4423 - val_accuracy: 0.7981 Epoch 96/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4470 - accuracy: 0.8029 - val_loss: 0.4813 - val_accuracy: 0.7681 Epoch 97/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4341 - accuracy: 0.7962 - val_loss: 0.4718 - val_accuracy: 0.7775 Epoch 98/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4639 - accuracy: 0.7906 - val_loss: 0.4529 - val_accuracy: 0.7881 Epoch 99/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4377 - accuracy: 0.8030 - val_loss: 0.4594 - val_accuracy: 0.7831 Epoch 100/100 320/320 [==============================] - 1s 2ms/step - loss: 0.4466 - accuracy: 0.7991 - val_loss: 0.4526 - val_accuracy: 0.7894
Plotting the train and test loss
# Capturing learning history per epoch
hist = pd.DataFrame(history.history)
hist['epoch'] = history.epoch
# Plotting accuracy at different epochs
plt.plot(hist['loss'])
plt.plot(hist['val_loss'])
plt.legend(("train" , "valid") , loc =0)
<matplotlib.legend.Legend at 0x7fe3cef7b210>
Lets evalute the second model against the test data
score = model2.evaluate(X_test, y_test)
63/63 [==============================] - 0s 2ms/step - loss: 0.4505 - accuracy: 0.7940
## Confusion Matrix
import seaborn as sn
y_pred1 = model2.predict(X_test)
for i in range(len(y_test)):
if y_pred1[i]>0.5:
y_pred1[i]=1
else:
y_pred1[i]=0
cm2=confusion_matrix(y_test, y_pred1)
labels = ['True Negative','False Positive','False Negative','True Positive']
categories = [ 'Retained','Exited']
make_confusion_matrix(cm2,
group_names=labels,
categories=categories,
cmap='Blues')
#AUC ROC curve
from sklearn.metrics import roc_auc_score
from sklearn.metrics import roc_curve
DNN_roc_auc = roc_auc_score(y_test, model2.predict(X_test))
fpr, tpr, thresholds = roc_curve(y_test, model2.predict(X_test))
# calculate the g-mean for each threshold
gmeans = np.sqrt(tpr * (1-fpr))
# locate the index of the largest g-mean
ix = np.argmax(gmeans)
print('Best Threshold=%f, G-Mean=%.3f' % (thresholds[ix], gmeans[ix]))
plt.figure()
plt.scatter(fpr[ix], tpr[ix], marker='o', color='black', label='Best')
plt.plot(fpr, tpr, label='Model 1 (area = %0.2f)' % DNN_roc_auc)
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic')
plt.legend(loc="lower right")
plt.savefig('ROC')
plt.show()
Best Threshold=0.514440, G-Mean=0.778
y_pred1 = model2.predict(X_test)
for i in range(len(y_test)):
if y_pred1[i]>0.514440:
y_pred1[i]=1
else:
y_pred1[i]=0
cm2=confusion_matrix(y_test, y_pred1)
labels = ['True Negative','False Positive','False Negative','True Positive']
categories = [ 'Retained','Exited']
make_confusion_matrix(cm2,
group_names=labels,
categories=categories,
cmap='Blues')
True Positive (observed=1,predicted=1):
Predicted Exiter and the customer is actually a Exiter.
False Positive (observed=0,predicted=1):
Predicted Exiter and the customer is not a Exiter.
True Negative (observed=0,predicted=0):
Predicted not an Exiter and the customer is not an Exiter.
False Negative (observed=1,predicted=0):
Predicted not an Exiter and the customer is a Exiter.
Important Metric¶The important Metric is recall. The bank will want keep as many customers as possible which means we want the least amount of False Negatives. Other words target the as many people as possible who are predicted to leave. With this being said, we also need to balance the False Positives as we don't want to spend too much money on marketing. The focus should be on recall sense it minimizes False Negatives
Country, Age and balance are important features with this data
The analysis used 11 independent variables. The first DNN model used a hidden layer with 8 nodes making use of relu for hidden layer and Sigmond for output layer. The initial accuracy of model was 86% with recall of 48%. Since the recall was low or false negatives were high, ROC threshold was optimized with a geometric mean of sensitivy and precision. Once a new threshold was found, the recall on model improved from 48% to 76%.
Even though this was a big improvement, a model 2 was built using class weights to handle the imbalance data. Also, increased hidden layers to two. The second model accuracy was 79% and recall was 74% without using theshold optimization. This was a substantial improvement in recall compared to the first model without thresold optimization. The second model with optimized threshold had accuracy of 80% and recall of 74%. Overall, there wasn't dramatic improvement between the model. With only 10k observations of data it is hard to develop the best model. I would recommend to try other methods such as logistic regression or a ensemble method. Also, I recommend gather more data